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Executive Summary 


The project 


The ReflectED programme was developed by Rosendale Primary School to improve pupils’ 
metacognition—their ability to think about and manage their own learning. This includes the skills of 
setting and monitoring goals, assessing progress, and identifying personal strengths and challenges. 
ReflectED consists of 28, weekly, half-hour lessons, which teach pupils strategies they can use to 
monitor and manage their own learning. Pupils are supported to apply and practise these strategies 
throughout the rest of the curriculum; reflect on their learning; and record audio, photographed and 
written notes of their reflections on Evernote, a note-taking app. Pupils are then encouraged to review 
and reflect on these records over time, so that they can observe their progress and consider which 
strategies seemed to work well. Teachers can also look across these records to get an overview of the 
areas that pupils are enjoying or struggling with, and identify specific pupil needs. For example, a 
teacher could explore the notes that a pupil has tagged as “maths” and “difficult” to see which ones 
they struggled with, and examine which strategies seemed to help them with this. 


In this project, Rosendale Primary School trained teachers from 30 schools in five areas throughout 
England to deliver ReflectED over the academic year 2014/15. At the beginning of the year, 
participating teachers received a pack of lesson plans and supporting resources, and an initial day- 
long training session. This was followed by three additional half-day training sessions throughout the 
year. A website, digital resources, and weekly reminders and tips were provided by the London 
Connected Learning Centre. The National Education Trust supported school recruitment and test 
administration. The programme was co-funded by the Education Endowment Foundation (EEF), 
KPMG Foundation and Nominet Trust, and was part of a funding round focused on digital technology. 


The impact of the programme on the attainment of pupils in Year 5 was evaluated using a randomised 
control trial involving 1858 pupils. Year 5 teachers within each of the 30 schools were randomly 
allocated to either participate in the programme or to a control group which continued with their usual 
teaching. The primary outcome measure was pupils’ maths attainment. The evaluation also examined 
the impact on pupil’s reading attainment and attitudes towards reading and maths, and the impact on 
the maths attainment of pupils eligible for free school meals. Class observations, interviews and focus 
groups were conducted to examine how the programme was implemented and adapted by teachers, 
explore activity in the control group, and identify factors that might affect the impact of the programme. 
The close involvement of the original developer in the delivery of the programme means that this was 
an efficacy trial. Efficacy trials aim to test whether the intervention can succeed under ideal conditions. 


Key conclusions 


1. Pupils who participated in ReflectED made an average of four months’ additional progress in 
maths compared to pupils who did not. 


2. Pupils who participated in ReflectED made an average of two months’ less progress in reading 
compared to pupils who did not. 


3. The findings for the schools in this trial have moderate to high security. However, the analysis 
conducted suggests that we cannot conclude from this trial alone that the intervention would 
have a similar impact in other schools. 


4. Most schools were already teaching metacognitive and reflective skills similar to those 
encouraged by ReflectED. This might have limited the additional impact that ReflectED had on 
teachers’ practice and pupils’ outcomes. 


5. Teachers suggested that ReflectED would work best as a whole-school programme, and that 
they could deliver the programme more effectively after the first year of delivery. Future research 
could examine the impact of implementing ReflectED across all year groups in the school and 
allowing more time for the programme to become embedded. 
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How secure are the fi ndings? Security rating awarded as part of 
. the EEF peer review process 


The findings for all pupils involved in this trial have moderate to high security. Both the design and 
analysis were appropriate and well-conducted, and pupils who received the intervention were similar 
to the pupils in the comparison group. There was no evidence that activity in the control group was 
changed by the delivery of the programme in the intervention group. One padlock was removed from 
the security rating because more than 10% of pupils did not complete all of the necessary tests and 
were not included in the final analysis. 


What are the findings? 


In this trial, pupils who participated in ReflectED made an average of four months’ additional progress 
in maths compared to pupils who did not. Pupils who participated also developed a more positive 
attitude towards maths compared to the control group. On average, FSM-eligible pupils made two 
months’ additional progress in maths, but it should be noted that this result is less secure than the 
findings for all pupils. All pupils who participated in ReflectED made two fewer months’ progress in 
reading and developed a slightly less positive attitude towards reading compared to the control group. 
The evaluator tested whether these results provide good estimates of what would happen in other 
schools using a widely-used statistical method. According to this test, we cannot conclude from this 
trial alone that the intervention would have a similar impact in other schools. 


The process evaluation suggested that most schools were already teaching metacognitive and 
reflective skills that are similar to those encouraged by ReflectED, and this is likely to have continued 
in the control group classes. This might have limited the impact that ReflectED had on teachers’ 
practice and pupils’ outcomes. However, many teachers praised the systematic approach that 
ReflectED brought to their attempts to develop metacognition. The majority of teachers reported that 
the programme and materials were useful, and that pupils responded well. Few teachers appeared to 
make major adaptations to the programme. Some teachers reported that they found it hard to find time 
to fit in all of the lessons and struggled to use the database of reflections to inform their teaching. 


Teachers suggested that ReflectED would work best as a whole-school programme. This would 
enable the school to more easily embed the programme in its work; tackle issues with timetabling; and 
enable children to develop a reflective approach to learning earlier in school, so they have time to 
build on it throughout primary school. Teachers also suggested that it would be easier to effectively 
implement the programme once they had delivered it for the first time, indicating that the programme 
might have a greater impact after the first year of delivery. 


How much does it cost? 


Assuming that a school has the tablet or laptop computers required to record reflections on Evernote 
and delivers the programme over three years, the annual cost of the programme would be £18.72 per 
pupil. This includes the cost of training, photocopying, and the license to use Evernote. Participating 
teachers were required to attend two and a half days of training over the course of the year. 


Table 1: Executive Summary Table 


Number of eet ize Estimated EEF security EEF cost 
Outcome (95% months 


Coxeyayitei=varex-)) progress 


Yo) afore] rating 


0.30 


(-0.04,0.63) ae 


. -0.15 
Reading (-0.59, 0.29) -2 months 
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1. Introduction 


Intervention 


This report describes an evaluation of ReflectED, which aims to support primary-aged pupils to 
develop their metacognitive or ‘learning-to-learn’ skills. Improving children’s metacognition focuses on 
encouraging them to reflect on how they learn, then encouraging them to develop strategies to 
improve these learning processes and motivate them to use them (Meyer, 2010). Metacognition is 
widely believed to have a positive impact on children’s development and most studies suggest positive 
outcomes (see Background Evidence below). 


Teachers at Rosendale Primary School developed the ReflectED approach over a number of years. 
The ReflectED approach comprises the following: 


1. Pupils receive a weekly ReflectED lesson from their teacher who follows a series of lesson 
plans. 

2. Atleast once a week, pupils are expected to reflect individually on their learning in other 
lessons and record these reflections on the Evernote platform (accessible through any tablet, 
netbook, laptop or PC). The lesson plans include tasks for the week, to support pupils to 
practice their metacognitive skills throughout their normal lessons. 

3. Children code their reflections to record their thoughts on a lesson and their performance. 
This enables them, and the teacher, to read previous reflections to inform future teaching and 
learning. 


A full description of the ReflectED approach can be found at 
http:/www.ReflectEDlearning.org.uk/about-ReflectED/. 


The Education Endowment Foundation funded Rosendale to develop a programme including training, 
support and materials that could help other schools to adopt the approach. There are a total of 28 
lessons in the materials provided by Rosendale. These lessons teach children specific metacognitive 
strategies they can use to support their learning, and teach them to reflect on their learning. For 
example, in the first lesson children are encouraged to think about when they learned something new 
and are then taught to juggle or do origami, which is not a skill they have learned before. Throughout 
this lesson pupils are encouraged to talk about how instruction helps them to learn, about learning in 
stages and the importance of practice. 


Enabling children to reflect well takes time and is scaffolded using a number of tools. A colour-coded 
system is used for the children to quickly show how well they thought they had done in a particular 
task. Green, for example, is used by a child to show they have been successful at the task and blue if 
they believe they have mastered the topic. Yellow denotes they are struggling and red is used to show 
that they are ‘stuck’. A set of pictures showing different feelings, like happy faces or rain clouds, is also 
provided along with a range of emotion words for the children to express their feelings about the 
learning process. The colour code, pictures and words are displayed in the classroom for children to 
quickly refer to. In the early stages of the process children fill in paper-based templates, which are 
pasted into their workbooks. When children start using the Evernote software on the iPads to record 
their reflections, they can either take pictures of the paper-based templates, or take pictures of the 
images or words expressing emotion and add a typed comment. These comments can become very 
detailed. They can also add audio or video to record reflections. A series of key words (‘tags’) are 
used to label the posts on Evernote so that they can be easily searched either by the children or the 
teacher. Children and teachers can use these tags to go back and review their progress and find 
areas that need more work. 


Some ReflectED lessons focus on getting the children to develop their skill in using the Evernote 
software to record their reflections. As the year progresses teachers encourage pupils to use Evernote 
to record their reflections on a regular basis, ideally several times a week, across different subjects. 
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A team of mentors from Rosendale Primary School trained and supported the primary schools that 
were part of the trial. Schools delivered the intervention to their Year 5 pupils, supported by the mentor 
teachers from Rosendale and online support from the London Connected Learning Centre 
(http://londoncic.org.uk). 


A well-attended launch day took place in summer 2014 at Rosendale School where teachers were 
introduced to the ReflectED approach. Participating schools were organised around five hubs: 
Sheffield, Gloucestershire and the Midlands, Hertfordshire, and two hubs in London. Following the 
launch day, there were three hub meetings where groups of the regional schools were brought 
together for half a day. One hub meeting took place in autumn 2014 and two in spring 2015. These 
meetings covered key features of the ReflectED approach, reviewed upcoming lesson plans to make 
sure teachers were familiar with the materials and addressed any problems that had been raised by 
teachers taking part in the intervention. Contact with all the schools also included half-day visits by the 
assigned Rosendale school mentor. In addition, weekly reminders and tips were sent out from the 
London Connected Learning Centre (LCLL), an organisation that supports schools in setting up digital 
technologies to enhance learning. LCLL also developed a website 
(http:/www.ReflectEDlearning.org.uk/) and videos to support implementation. The programme 
materials are freely available online for any school to use. While the core materials remained the 
same, teachers did make adaptions to the process of teaching and recording of the reflections. For 
example, some teachers adapted the recording template, or agreed that these could be included in the 
pupil’s workbooks. These adaptations were discussed and agreed with the mentor teachers from 
Rosendale. Further information on adaptions is discussed in the process evaluation section of this 
report. 


The intervention evaluated in this trial lasted for one academic year. However, ReflectED is designed 
to promote learning as a continuous approach throughout primary school rather than a short-term 
intervention that is delivered as a ‘one-off. 


This project was funded by EEF in partnership with Nominet Trust, as part of their themed round on 
Digital Technology: https://educationendowmentfoundation.org.uk/news/eef-and-nominet-trust- 
announce-new-partnership. 


Background evidence 


ReflectED is informed by the research on metacognition. The study of metacognition was pioneered 
by Flavell in the mid-1970s (Flavell, 1979). Initially, the field had a particular focus on children’s 
memory, as the early experiments that Flavell and colleagues conducted to explore metacognition 
were based on how well children of different ages were able to recall a list of items. In the recent 
literature, metacognition is often defined as pupils’ ability to think about their learning explicitly, or 
‘learning to learn’. The term is frequently used in association with terms such as reflection (Epstein, 
2003), thinking skills (Costello, 2012), self-reflective learning (Bandura, 1986), thinking intentionally, 
self-regulation (Kuiper & Pesut, 2004) and independent or autonomous learning (Luftenegger et al. 
2012). Metacognition is generally recognised to have two components: 


1. anunderstanding of what learning is and awareness of effective learning strategies; and 
2. the ability to select the most effective strategy for the current task. 


This combination of awareness and application is captured in Meyer et al.’s succinct definition: 
“Metacognition refers to the awareness, knowledge and control of cognition” (2010: 85). 


Most research on metacognition focuses on students in secondary, rather than primary, settings. This 
is possibly because students are required to work with a greater degree of independence as they get 
older. Some research has found that pre-schoolers or pupils in the early stages of primary school 
possess a very basic level of metacognition, and that it is only towards the end of primary school that 
metacognition becomes more sophisticated and academically oriented (Veenman, Hout-Wolters & 
Afflerbach, 2006 in Mevarech, 2010). These theories build on Flavell’s original assumption that the 
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younger the child, the more limited their memory capacity, and, by corollary, their metacognitive ability 
(1979). 


Research on the value of metacognition 


This section describes the research on the impact of metacognition in primary contexts. It is widely 
acknowledged by educationalists and researchers that metacognition underpins much successful 
learning (Israel et al., 2006). Conversely, a lack of metacognitive skills puts students at a disadvantage 
in the classroom (Joseph, 2009). Many researchers describe benefits similar to those initially claimed 
by Flavell: 


...metacognition plays an important role in oral communication of information, oral persuasion, oral 
comprehension, reading comprehension, writing, language acquisition, attention, memory, problem 
solving, social cognition and various types of self-control and self-instruction.... (Flavell, 1979: 906) 


Some studies have found that there is a direct link between metacognition and academic ability. The 
Sutton Trust-EEF Teaching and Learning Toolkit finds that pupils involved in interventions aimed at 
improving their metacognition make an average of eight additional months’ progress over the course 
of the year. Students with good metacognition have been found to perform better academically than 
those with poor metacognition (Coutinho, 2008). Metacognitive skills have been identified as one of 
the most important predictors of educational achievement (Bryce et al., 2014) and one of the most 
important factors affecting learning (Mok et al., 2006). 


A recent study of primary-aged children learning English as a foreign language in Portugal explored 
whether training to regulate learning would affect how students report self-regulated learning in 
diaries. The study followed an experimental group of 40 and a control group of 60. Students’ academic 
achievement was also assessed via a diary study through which children were trained to reflect on 
their learning following oral and vocabulary tasks. The findings suggested that those students who 
experienced the training produced higher-quality reflections in their diaries and demonstrated better 
academic performance on language tasks than their peers in the control group (Costa Ferreira et al., 
2014). 


It has also been suggested that developing the range of metacognitive tasks may have a positive 
impact on student motivation (Sungur et al., 2009). Sungur et al’s study showed that in the particular 
context of Turkey, a system with lots of exams, pupils were only motivated to employ a very limited set 
of strategies in order to pass the exams and move on to higher education. However, if they were given 
novel and more challenging tasks their motivation towards different strategies increased. 


The age of the pupils may be a factor that determines the effectiveness of metacognition instruction, 
with younger pupils benefiting more than older pupils (Mevarech, 2010). A study based in Israel found 
that younger students in Grade 3 benefited more from a metacognition intervention than their 
counterparts in Grade 6 (Mevarech, 2010). The study suggests that because children are not aware 
enough of what they know or when to apply particular strategies it is important to start early in order to 
encourage children to get used to the idea of reflecting on their learning and applying particular 
strategies. 


However, not all researchers are convinced of the academic benefits of using metacognition in the 
classroom. Sperling et al. highlight that “many studies report little relation between metacognition and 
achievement or aptitude in children” (Sperling, 2012: 2). Other researchers have pointed to a lack of 
reliability in the data collected, much of which may have been compromised by pupils self-reporting 
data in media such as diaries or journals (Lee, 2012; Serra & Metcalfe, 2009). 


The ReflectED approach was devised by and initially piloted in Rosendale Primary School. Early 
results reported by Rosendale School suggest that the approach has had an impact on children's 
attainment. In the first Year 3 class to experience the ReflectED approach, children eligible for free 
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school meals (FSM) made 4 Average Point Score (APS) of progress in maths and reading during the 
year, compared to the 3 APS of progress made by FSM children in the other Year 3 classes. 


The trial reported here examined the efficacy of the ReflectED approach and responds to calls for 
more work to be undertaken in showing whether metacognition can have impacts on numeracy and 
literacy (DFE, 2012). Numeracy was chosen as the primary outcome for the trial following consultation 
between the evaluation team, EEF and Rosendale. 


Evaluation objectives 
The principal research question is: 

e What is the impact of the ReflectED approach on progress in maths? 
Secondary research questions included: 


e What is the impact of the ReflectED approach on reading? 
e What is the impact of the RefelectED approach on pupils’ attitudes to maths and reading? 


Other questions connected to the process evaluation include: 


What value do the schools see in the ReflectED approach? 
What are the teachers’ attitudes towards the intervention? 
What are the learners’ attitudes to the approach? 

What do teachers and learners perceive the benefits to be? 
Is the approach scalable? 

What are the barriers to delivery? 


Project team 


Kate Atkins, who is the head teacher of Rosendale Primary School, ran the school-based project. She 
was supported by Marc Rowland of the National Education Trust, who was the project manager and 
dealt with day-to-day issues from school recruitment and supported the delivery of tests. Joe Halloran 
from the London Connected Learning Centre and a team of teacher mentors from Rosendale Primary 
School supported the project throughout, running training and technical support for schools. 


The evaluation was led by Gary Motteram with support from Graeme Hutcheson who was responsible 
for designing and managing the trial design and the initial quantitative data analysis. The process 
evaluation was overseen by Zeynep Onat-Stelma with assistance from Afroditi Kalambouka and 
Joanna Bragg. Sophina Choudry ran the final quantitative analysis and co-authored the final report. 


Ethical review 


The project underwent a rigorous ethical review process in the University of Manchester. The approval 
reference number is 14225. 


The main ethical issues raised concerned obtaining consent from the participating schools, staff and 
pupils as well as the protection of confidentiality and anonymity of the schools, teachers and other 
school staff and pupils. 


All schools interested in participating in the trial attended initial ReflectED events, which provided 
information about the project and supported recruitment to the study. Schools were able to discuss the 
details of the programme and study with staff from both the Rosendale School and the University of 
Manchester evaluation team. Written consent was also gained from all participant teachers in the 
school case studies and/or subsequent interviews in follow up events. All pupils in participating 
classes took part in the intervention as this was seen an inseparable part of the pupils’ school day/life. 
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All parents received carefully written information about the project. An opt-out parental consent 
process was followed, allowing parents sufficient time to make a decision about whether pupils should 


take part in the testing of outcome measures and focus groups. All researchers who visited schools 
underwent enhanced DBS checks. 


Protection of confidentiality and anonymity followed the formal procedures of the University of 


Manchester (http:/Mwww.manchester.ac.uk/research/environment/governancelethics/) through 
password protection and encryption of electronic documents and devices. 
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2. Methods 


Trial design 


The research design was a school-based randomised controlled trial with class-level randomisation. 
There were two arms: intervention (ReflectED) and control (business as usual). The plan in the 
protocol had been to recruit 24 schools but in the final analysis 30 schools were involved in the trial 
because Rosendale School had managed to get more interest from local clusters. The increased 
number of schools increased the power of the study. 


Class-level randomisation was chosen in preference to other forms of randomisation for a number of 
reasons. First, class-level randomisation offered slightly greater efficiency in terms of statistical power 
than school-level randomisation. Second, class-level randomisation reduces the likelihood of post- 
allocation resentful demoralisation and attrition since all participating schools are able to access the 
intervention. Third, ReflectED was designed as a whole-class intervention, making individual-level 
randomisation impractical. However, class-level randomisation also carries an increased risk of 
diffusion/contamination, which is a threat to internal validity. Steps were taken to prevent this, with 
clear guidance provided to participating schools about the importance of maintaining the integrity of 
the ‘intervention’ and ‘control’ arms of the trial. Furthermore, the potential for contamination was 
monitored as part of the process evaluation (see below). 


Outcome measures 


CEM’s InCAS computer adaptive tests were administered online towards the end of the intervention, 
in June 2015, nearly an academic year after the start of the intervention. A training day took place in 
Rosendale School to make sure that the Rosendale mentors understood how the test is administered, 
and what some of the technical issues might be. For example, Rosendale ensured that the mentors 
understand how the children should log in and how to solve access through school firewalls. Each 
school became responsible for administering their own tests to pupils with support provided from both 
Rosendale and the University of Manchester team. Schools split pupils into groups, which took the 
tests at different times over a two-week period. We were therefore unable to know exactly when the 
tests would be taken and could not provide independent monitors during the testing period. We 
recognise that this might impact on validity, but the need to allow the pupils to complete the tests 
independently was stressed in the briefings by the mentor teachers from Rosendale and when the 
mentors visited the schools they checked that this was being adhered to. The mentor teachers visited 
all of the schools and there was no direct evidence that this was a threat to validity. The University of 
Manchester team worked with the London Connected Learning Centre to provide further technical 
support. 


Scores from the tests were accessed through the online secure CEMs site. CEMs tests are routinely 
used in evaluation research and RCTs in particular (see http:/Mwww.cem.org/evaluation) and offer 
age-related standardised scores in each of the modules (except for the attitudinal measures). 


Primary outcome 


The primary outcome is the age standardised mathematics score. 
Secondary outcome 
e Standardised Reading scores 


e Standardised Attitude to Mathematics scores 
e Standardised Attitude to Reading scores 
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Baseline Test 


Key Stage 1 Mathematics score accessed via the National Pupil Database (NPD) was used to 
measure children’s academic attainment at age 7 and used as a baseline for the analyses on pupils’ 
maths attainment and attitudes to maths. KS1 reading scores were used as a baseline for the 
secondary outcome analysis of InCAS Reading and attitude to reading scores. We have included prior 
attainment to have a value-added model that measures academic progress. Such a model increases 
statistical power and reduces estimate bias of the effect sizes in the analysis. 


Participant selection 


Potentially all mainstream primary state schools in England with at least two classes of Year 5 pupils 
were eligible for the study. However, as the intervention relies on technology use, a criterion for having 
suitable technology to run Evernote and effective wireless connection was also added. Good internet 
connectivity was also required for conducting the CEM InCAS test and all schools did have good 
enough access. However, none of the schools had enough computers to test a whole class in one 
batch so the children were tested in smaller groups. The following eligibility criteria were set in 
recruiting schools: 


1. Schools must have at least two forms of entry (larger schools can also participate) 

2. Schools must have an effective wireless network 

3. Schools must have access to enough in-class technology to provide access to Evernote. 
iPads were recommended. 

4. Schools must have a minimum 10% of children in receipt of Pupil Premium funding. 

5. Schools must not have ability setting for Year 5 pupils (if they currently ‘set’ pupils in Year 5, 
they should be prepared not to for the duration of their involvement in the project) 

6. Schools must agree and make an effort to be fully committed to all aspects of the project for 
the duration and sign a Memorandum of Understanding 


Schools were recruited via LAs or via lead schools in a cluster. Several locations responded positively 
and as a result the delivery team ended up focusing on a number of different locations in London, but 
also in Sheffield, Hertfordshire, the Midlands and Gloucestershire. Having schools grouped in specific 
areas rather than all over the country would make it more time and cost effective for: a) Rosendale to 
provide training and support, b) for the research team to collect data and provide other support, and c) 
schools to form networks to support each other. 


The Rosendale team contacted a small number of schools directly, as well as speaking to groups of 
headteachers through local authorities or cluster leads, e.g. Gloucester School Partnership, Kingston- 
upon-Thames headteachers’ group meeting. Eligibility criteria were stressed from the very beginning. 
Only two schools that were approached declined to take part, one because of the RCT design and one 
because of a recent bad Ofsted report. Of the 33 remaining, three were excluded because they were 
one form entry. These three schools came to the training and taught the lessons, but were not a part 
of the RCT. 


Following the initial recruitment meeting, all parents of children involved in the study were sent a letter 
informing them of the intervention and the evaluation. Letters were given to the pupils through their 
respective schools (see Appendix 1). Parents were given an opt-out for their children’s participation in 
the research project. This meant that children in treatment classes would still be part of the 
intervention but would not participate in the evaluation (whether this was the focus group interviews or 
sitting the InCAS tests at the end of the project). Individual children in treatment classes were not 
given the opportunity to opt out from the intervention per se as this was seen as potentially beneficial 
and not deviating from the school’s normal day; also since the intervention would ideally be embedded 
in the whole curriculum, opting out would be unsettling both for the pupil, their peers and the teachers. 


Education Endowment Foundation 12 


ReflectED 


Class lists and UPNs were collected from the schools by the University of Manchester team with 
support, where necessary, from colleagues in the Rosendale team. We collected a complete set of 
FSM data from the NPD. 


Sample size 


After reviewing the sample size in collaboration with EEF, it was agreed that 24 schools would be 
recruited in a number of locations and with a number of types of school represented. As was 
mentioned earlier 30 schools were in fact recruited because of interest in the project and this had the 
impact of increasing the power of the study. The following assumptions were taken into account in the 
power calculations: 


1. Aclass-level intra-class correlation (ICC) of 0.133 (for the primary outcome based on a school 
level ICC, see Hedges & Hedberg, 2007). 

2. Equal cluster sizes of 30 pupils per class. 

3. 60 classes participating in the trial (30 schools assumed to have 2 classes in each school). 

4. 14% of the variance in the post-test (primary outcome) is assumed to be explained by prior 
attainment (based on school level variance, see Hedges & Hedberg, 2007). 

5. Power and significance to be at 0.8 and 0.05 respectively. 


Based on these assumptions, the trial analyses would have been able to detect an intervention effect 
size of 0.27 standard deviations or greater. 


In the actual sample, a total of 1858 pupils over 70 classes were included. Thus, steps 1 to 5 can be 
updated as follows for the randomised sample: 


A class level intra class correlation (ICC) of approximately 0.11 (for the primary outcome). 
Average cluster sizes of 27 pupils per class. 

70 classes with 38 allocated to the intervention group and 32 to the control group. 

28% of the variance in the post-test (primary outcome) explained by prior attainment. 
Power and significance to be at 0.8 and 0.05 respectively. 


arRWN > 


The Minimum Detectable Effect Sizes (MDES) at randomisation is therefore 0.3. Further details of 
MDES can be seen in Table 4. 


Randomisation 


Schools in the study were a mix of two-, three- and four-form entry, so it was agreed that all Year 5 
teachers would take part. Teachers/classes were randomly allocated to the treatment group and 
control groups. Teachers were linked to a class of pupils before randomisation. No changes were 
allowed once teachers had been allocated to their group. 


Random allocation to treatment and control groups was achieved using a computer randomisation 
algorithm. 


There were a total of 30 schools in the sample including 2 four-form, 6 three-form and 22 two-form 
entry schools. In order to make experimental pairs, classes were ordered within schools in 
alphabetically. The 2 four-form entry schools were split into two pairs each, that is, four classes within 
each of the four-form entry schools were treated as two pairs resulting in 4 pairs in total (8 classes). 
Thus, there were 26 experimental pairs plus 6 three-form entry schools. 
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The 26 experimental pairs were allocated to the treatment or control group using a simple random 
sample procedure available in the R package (R Core Team, 2015). 0 and 1 were randomly generated 
and then applied to the experimental pairs — O indicating the control group and 1 indicating the 
treatment group. The following code was run once for each selection made. 


sample(0:1, 1, replace=T) 


Similarly, the random selection of the control class for the 6 three form entry schools was made using 
the same computer randomisation algorithm. Selecting a control group from three classes is similar to 
the two classes procedure: 


classes <- c("class01", "class02", "class03") 
sample(classes, 1) 


Thus, one class out of the 3 classes in each of the 6 three-form entry schools was randomly assigned 
to control. Random selection was applied by one of the researchers, who applied the selection 
procedure without any knowledge of the school or classes. 38 classes were assigned to treatment and 
32 to control. Although this procedure led to unequal allocation, this only has a substantive effect on 
power when the ratio is 3:1 or more (Dumville et al, 2006). 


During the process evaluation visits, teachers were specifically asked about randomisation and 
confirmed that their class had been selected randomly to take part in the project. However, the 
process evaluation did not include all the schools. 

Analysis 

We applied a two-level multi-level model, where pupils were clustered at the class level. Additionally, 
we also fitted the condition (i.e. ReflectED intervention, with control as the reference group) at the 


individual pupil and class level to reflect the design of the trial: treatment or control was administered 
to complete classes (i.e. randomisation took place at class level): 


Yio" =at B.Cond) +y.Vii° + 6.Xi + &y 
Where: 
yPost = standardised InCAS mathematics scores (for the primary outcome analysis) 
yPre = Key Stage 1 mathematics attainment scores (for mathematics based InCAS measure) 
X = control variables (i.e. Free School Meal eligibility) 
€ = error term for pupils clustered at class level 
i = pupil i 
j = class j 
Standardisation 


In order to compute Hedges G, standardisation of response variable data was undertaken in order to 
produce a standardised co-efficient in the model (please see Jerrim et al (2015) for an example of an 
EEF evaluation following a similar process). Thus, effect sizes (Hedges G) were calculated by firstly 
converting the raw scores into standardised Z scores: 
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z, < Mun) 
- SD pooled 


Where: 

M = pupils’ InCAS mathematics scores (for the primary outcome analysis) 

M = the whole sample mean InCAS mathematics score (for the primary outcome analysis) 
SDpoo = pooled standard deviation used to calculate Hedge’s G. 


The pooled standard deviation was calculated as follows: 


[ou-D+s}+@m-D=s3 


SD pooled = Mm +N —2 


Where 


n1 = number of pupils in the treatment group 


n2 = number of pupils in the control group 


s1 = standard deviation of InCas mathematics test scores in the treatment group (for the primary 
outcome analysis) 


$2 = standard deviation of InCas mathematics test scores in the control group (for the primary outcome 
analysis) 


Missing Data 


Missing data was predominantly at the school level with two schools providing no data on all outcome 
variables and/or co-variates. Two further schools provided no data on reading and one on 
mathematics (see table on descriptive statistics). As the current dataset has missing values 
(intervention — 16% and control — 15%, including school attrition and other reasons such as pupil 
absence on the day of the tests or incomplete tests) and these cases are greater than the 5% 
threshold, as per EEF’s guidelines, we have carried out a sensitivity analysis, as well as multiple 
imputations for the MLM models. Multiple imputation procedures were carried out using the software 
REALCOM-IMPUTE assuming that the data is missing at random (Carpenter, Goldstein and Kenward, 
2011). Thus we were able to include partially observed cases (i.e. cases that have either the InCas 
mathematics score or the KS1 mathematics test score missing) of all 1858 pupils in the analysis and 
reduce bias. Such an approach is in line with the EEF protocol for intention to treat analysis. 
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Table 2: Reasons for missing data 
Missing Data Taiccyarcclaitcelame| cele o) Control group 


z Percentage ) Percentage 
Reasons (Ke) g (Ce) g 


{1) {1} 
intervention) i) control) ko) 


ms 61 
Absence or other 
reasons 


Total missing 


15 


Protocol, registration and data availability 


The project protocol can be obtained here: 
https://educationendowmentfoundation.org.uk/public/files/Projects/EEF_Project_Protocol_Refl 
ectED.pdf 


The trial was registered post hoc: ISRCTN41017069 
Implementation and process evaluation 


The process evaluation consisted of three main phases: 


1. Atthe beginning of the project process, the University of Manchester team were involved in a 
number of the recruitment events as observers. 

2. The Manchester team visited 20 schools in the middle of the academic year. The team 
observed both treatment and control classes using a standard framework for noting time and 
activity, interviewed the intervention teacher and held a focus group of children from each 
school. We approached all of the schools to take part in the observation process and worked 
with any that responded. We made sure that there was a representation from the three main 
clusters of schools: London; Oxfordshire and Gloucestershire; and Sheffield. We also 
conducted telephone interviews with the treatment class teachers from other schools that we 
had not been able to visit. We did this so that we could capture data at a similar point of time 
with as many schools as we could. 


The observation of the lessons and the interviews that were conducted at the mid-point of the year 
were designed to establish: 


e Usual practice — views on the reason for engaging in the metacognitive approach and their 
perceptions on the need for it in the school, and teachers’ descriptions of their previous 
practice around metacognition. As will be seen below, this latter question has serious 
implications for the trial as conducted. 

e Fidelity — the extent to which the school adhered to the intended treatment model was judged 
by checking that the necessarily tools to support the children’s reflections were displayed in 
the classroom, that proformas were being used to capture reflection and that reflections were 
being transferred on to Evernote. As well as the observations and teacher interviews where 
we asked about practice, we also confirmed with the children in the focus groups the type of 
reflection that was occurring. 

e Adaptations — changes that teachers made as they implemented the programme. 

e Dosage and Quality -how much the intervention was included into the day-to-day practice of 
the schools and how well it was included. 
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e Participant responsiveness — what the reaction of both the teachers and the children involved 
in the project was to the materials, process and activities related to the implementation and 
the benefits and enjoyment of the groups of learners. 


3. Atthe end of the school year, the Manchester team attended the final celebratory event as 
observers, but also to conduct focus groups with the 28 schools that attended this event. 
These focus groups were in lieu of the originally planned final questionnaire and enabled us to 
follow-up themes that were emerging from the qualitative analysis in a way that 
questionnaires would not have done. They also allowed us to triangulate some of the 
emerging findings from the themes from the observations, interviews and children’s focus 
groups. 


The team also conducted interviews with the mentor teachers in Rosendale and with Kate Atkinson, 
the head teacher, about their experiences and impressions of the intervention. This was to provide us 
with additional background data about the mentoring process and the mentors’ views of how 
successful they believed the project had been. 


Cost 


The cost calculation is based on discussions both with the teachers in the final focus groups in July, 
2015 and subsequently with the head teacher, Kate Atkins and her team at Rosendale. 


Timeline 


Table 3: Timeline 


Date Activity 


Preparation and analysis of the qualitative data (transcription of 
interviews, writing up of observations and case studies, initial coding and 
analysis using NVivo package) (MIE) 


Preparation and analysis of the quantitative data (MIE) 


ReflectED celebration event (Host: Rosendale; Attendance: School & 
MIE). Collection of focus group data. 
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Figure 1: Flows of schools and children through the trial (primary outcome analysis) 


Approached School n=36 


Lost to follow up: 
classes n=3; 
pupils n=84 


Missing data: 
Tests not 
completed due to 
absence, etc.: 
pupils n=74 


Assessed for eligibility 
School n=33 


Declined to participate 
School n=3 


Randomised 
classes n=70; pupils n=1858 


Allocated to 
intervention 
classes n=38; 
pupils n=997 


Post-test data 
collected 
classes=35; 
pupils n=913 


Total 
analysed 


classes n=35; 
pupils n=839 
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Allocated to 
control classes 
n=32; pupils 
n=861 


Post-test data 
collected 
classes = 30; 
pupils n=800 


Total analysed 
classes n=30; 
pupils n=731 


Not meeting inclusion 
criteria School n=3 


Lost to follow 
up: class n=2; 
pupils n=61 


Missing data: 
Tests not 
completed due to 
absence, etc.: 
pupils n=69 
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Table 4: Minimum detectable effect size at different stages 
N Correlation [exes Power Alpha | Minimum 
[classes/pupils] between pre- fo(=\ req rl 0) (=) 
(n=intervention; test & post- effect size 


ra rexe) a) dae) )) test (1b) $59) 


60/1800 


70/1858 


65/1570 


Table 4 shows the MDES for the protocol, after randomisation and based on actual sample size post 
hoc InCAS tests. The MDES of 0.40 at the analysis stage indicates that the high ICC increased MDES 
in the final sample, even though a higher number of schools/classes were involved than initially 
planned at the protocol. The difference in ICC can be attributed to a number of factors. The estimated 
ICC at the protocol stage was based on existing literature. The variation in KS1 mathematics test 
scores between clusters is (as expected) low at 9%. However at the analysis stage, we have used 
INCAS mathematics scores and there appears to be great variation between classes (28%). The 
INCAS measure differs significantly from the KS1 data used for the estimates in earlier stages in that it 
is represents standardised test data as opposed to teacher assessment data. 


School and pupil characteristics 


Table 5: Baseline comparison of school and pupil characteristics 


Variable n Percentage 
School-level (categorical) 
School type 
Academy converter 2 7 
Academy sponsor led 3 10 
Community 18 60 
Voluntary aided 4 13 
Voluntary controlled 1 3 
Ofsted rating 
Outstanding 6 20 
Good 20 67 
Requires Improvement 3 10 
Inadequate 1 3 
School N 30 100 
Class N 70 100 
Pupils N 1858 100 
Variable Intervention group (oxelnlige) Me] celele) 
Pupil-level (categorical) n Pea n ey ad 
Eligible for FSM 334 33.6 314 36.6 
Not eligible for FSM 659 66 545 62.8 
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Missing (FSM) 4 0.4 2 0.2 
Pupil-level (continuous) freeway Mean feaiceing) Mean 
KS1 reading point score agi 16.1 Bi 16.0 
KS1 mathematics point 949 839 

score (48) oe (28) 2 
Classes N 38 32 

Pupils N 997 861 


As can be seen in Table 5, the predominant school type in this study was Community schools. 
However, academy schools comprised almost 17% of the total. Schools were mostly rated (%) as 
good with six being outstanding. 


Table 5 also compares the raw averages for pupils’ prior attainment in KS1. As evident, there is little 
variation between the control and treatment groups’ prior reading and mathematics attainment raw 
means (dksireading = 0.1; dksimaths = 0.1). This then suggests that the treatment and control groups are 
well balanced in terms of prior attainment. 


Finally, Table 5 compares the control and intervention (ReflectED) groups in terms of pupils’ 
background characteristics (for all 1858 students initially randomised — missing data has also been 
computed). As evident, there are fewer free school meal pupils (314 versus 334) in the control group 
compared to the treatment group. 


Outcomes and analysis 
Missing data 


As noted earlier, we used multiple imputation in order to account for missing data for the InCAS 
scores, as well as for covariates such as prior attainment and FSM eligibility. Doing so allowed us to 
include incomplete or partially observed cases, thereby enabling fidelity to intention to treat principles 
in our analysis. In the next sections, we present the findings of both the partially observed (imputed 
analysis) and complete cases side by side for comparative purposes. 


After obtaining additional data from the NPD, the missing observations for the covariate FSM were 
significantly reduced to 6 cases (see Table 5 for a complete overview for each of the variables). In 
summary, there are a total of 1521 complete cases (out of 1858 total cases) and, thus, 337 incomplete 
cases (18%) for the entire population. Out of these incomplete cases, 261 are missing InCAS 
mathematics scores (either due to school attrition or other randomly missing data - e.g. students 
absent on the day of the post-test), 49 missing KS1 mathematics test scores and 27 missing both 
scores (INCAS maths and KS1 maths). As the number of missing or incomplete cases is greater than 
5%, the EEF guidelines recommend either multiple imputation or a sensitivity analysis. At the same 
time, we acknowledge that at 18% incomplete cases, this is above the usually recommended 5% 
threshold. There is a danger that at this level of missing or incomplete cases, the multiple imputation 
analysis could lead to under estimation of the standard error. Therefore, we provide both a complete 
case analysis, as well as an imputed analysis for comparison (see Table 6). 


We also present a regression analysis for the primary outcome in order to investigate the missing 
data. In other words, we have introduced a binary variable for complete (1) and incomplete cases (0). 
Here, a complete case is defined as a case that has the post-test (INCAS maths score) and pre-test 
(KS1 maths score) present. We carried out a regression analysis with both complete and incomplete 
cases as an outcome variable and condition (treatment or control) and FSM eligibility as explanatory 
variables. We found that students eligible for FSM were more likely (but only by a slight margin) to 
complete the pre and post-tests (INCAS maths tests) with log odds = 0.24 leading to an odds ratio of 
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1.27 (please note that an odds ratio of 1 indicates an equal chance of FSM and non FSM students of 
having complete cases). Students who were assigned to the intervention group (log odds = -0.105, 
odds ratio = 0.9) were less likely to have complete cases. However, both of these effects were found 
to be non-significant (at p < 0.05). 


Primary Outcome 


Table 6: Effect sizes for Primary Outcome — with and without imputation 
Raw means Effect size 


Intervention group Ofol alice) Mel gele] e) 
n nin model 


Mea 
(95% Cl) 


n Mean aie 
Outcome (missin 


(missing) (95% Cl) ) 
InCAS Maths 


Score 800 707 1570 


(complete (158) (130) (800; 707) 0.04,0.63) 
cases) 


InCAS Maths 96.9 96.98 
Score with | Osa, | 281 asas, |g l8S8. | OT 
imputation) 98.33) 98.53) ; aed 


Table 6 shows that, firstly, that there is no substantial difference between the effect sizes of an 
imputed (0.27) and complete case model (0.3). It is also evident that there is no statistically significant 
difference between students who have received the ReflectED treatment and students who have not. 
The intervention effect size is moderate, (i.e. 0.3 with the confidence interval -0.04 to 0.63). 


Hedges g 


(interventio (95% Cl) 


la exe) a) 40) 
0.30 


Subgroup Analysis - FSM ONLY 


For the subgroup analysis, we created a data subset with only FSM eligible students (see appendix 2 
for the MLM models). 


Table 7: Effect sizes for Subgroup Analysis (Primary Outcome) - with and without imputation 
Raw means Effect size 


Intervention group fOfol alice) Mel geley e) 


n ami amanteye (=| 
Edita Mea ‘i : 
(missin (95% Cl) (interventio 

: la exe) a) 40) 


n Mean 
(missing) (95% Cl) 


Hedges g 


p- 
Outcome (95% Cl) WEL} 
InCas Maths 
Score : : 544 


(complete . ‘ (281; 263) 


InCas Maths 
Score (with 
imputation) 


From Table 7, it is evident that there is no statistically significant difference between students with 
FSM who have received the ReflectED treatment and students with FSM who have not. The 
intervention effect size is small. 
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Secondary Outcomes 


Table 8: Effect sizes for Secondary Outcomes - with and without imputation 
Raw means Effect size 


Intervention group Ofol alice) Me] cele e) 


n ami aminceye(e)| 


missin Mean interventio 
(95% Cl) 
aero) a) dco) 


n Mean 
(missing) (95% Cl) 


Hedges g 


Outcome (95% Cl) 


InCAS 

preci 726 627 1252 
(complete (237) (210) (726; 627) 

cases) 

InCAS 

Reading 997 861 1858 
Score (with (237) (210) (997; 861) 
imputation) 

InCAS 
| ne | Qt | san | 028 
(complete (t2) (28) ; (08,718) 0.24,0.71) 
cases) 


InCAS 
Attitude to 861 ; 0.35 
Maths Score Hk (120) tee (- 
(153) (997; 861) 
(with : : 0.17,0.86) 
imputation) 
InCAS 
Attitude to 
Reading 805 : Hoo) 1521 
Score (153) : ; (805; 716) 0.47,0.29) 
(complete : ; — 
cases) 
InCAS 
Attitude to ; 861 0.07 
. 997 1858 
Reading (153) : (120) (997: 861) (- 
Score (with : ; 0.36,0.49) 
imputation) 


The secondary outcome analysis (see Table 8) shows no significant differences in the standardised 
means between control and intervention students for INCAS reading, attitude to mathematics and 
attitude to reading scores after controlling for prior attainment and FSM. Intervention effect sizes are 
small, with the exception of the moderate effect size for attitude to mathematics. 


Cost 


The following cost information is based on both discussions with teachers in the final focus groups in 
July 2015 and information provided by the delivery team. 


It is expected that the majority of schools will have access to the IT required to implement the 
programme, so this cost is excluded from the cost estimate. The following costs are included in the 
cost estimate: 


e The cost of training, travel, project management and technology support. These costs are 
likely to be limited to the first year of implementation. This cost information was supplied by 
the developer. 
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e Photocopying costs for Reflection sheets (30 sheets per pupil per year). Photocopying is 
estimated to cost 6p per sheet, so the cost per pupil is £1.80. The total photocopying cost for 
the project was estimated to be £1,795. This cost would be repeated every year that a school 
delivers ReflectED. This cost information was collected through the focus groups with 


teachers. 


e The cost of the Evernote license. Evernote is free initially, but schools have to pay to acquire 
sufficient storage capacity for the large number of notes made by the pupils in the programme. 
This costs £34.99 a year, but one school only needs one licence. 


Table 9: Start up and ongoing cost information 


Cost type 


Cost item 


Project 
cost in 


Cost per pupil in 


first year 


first year 


Start-up cost Training £14,300.00 £14.34 
Start-up cost Travel £3,150.00 £3.16 
Start-up cost Technology support £20,000.00 £20.06 
Start-up cost Project management £10,000.00 £10.03 
Ongoing cost Evernote license (£34.99 per school) £1,049.70 £1.05 
Ongoing cost Photocopying £1,794.60 £1.80 

Total £50,294.30 £50.45 


The total cost of delivering the project in the first year to 30 schools was £50,294, or £1676 per school. 
Table 10 displays the cumulative cost per pupil over three years, and the average cost per pupil per 
year. The cost per pupil was calculated by dividing the total project cost by the number of pupils who 
were allocated to receive the intervention (997). 


Table 10: Pupil costs over three years 


Number of years 


Average cost per pupil per year (£) 


UTTate Wa tals COLTaaLUlr-VihV(cmexex-) my ol-1 am ol) o)] 629) (cumulative cost per pupill 
programme number of years) 
Year 1 £50.45 £50.45 
Year 2 £53.30 £26.65 
Year 3 £56.15 £18.72 


The average cost per pupil per year would be £18.72 over three years. 
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4. Process evaluation 


Here we provide an overview of selected findings from the qualitative component of the study, 
showing the implementation and process-related issues that emerged from our analysis. The selection 
of points represents the issues most germane to the findings presented earlier in the report. Data 
excerpts come from teacher interviews, observations and the focus groups both with the teachers and 
the children. All the interviews and focus groups were audio recorded and transcribed as necessary. 
The data was inductively coded and a set of themes was established by the evaluation team 


Context: Metacognition in schools 


It was clear from the analysis of the qualitative data set that metacognition/‘learning to learn’ is seen 
as an important component in school practice, as this was a common issue across all of the schools 
that were visited. Improving learning outcomes and learning behaviour for all pupils was the general 
consensus in terms of perceptions of needs, benefits and expected outcomes. 


Although teachers reported that they had not personally used anything as systematic and structured 
as the ReflectED approach in the past, it appears that nearly all teachers interviewed reported that 
they had been practising reflective learning in some way or another, or that they have introduced 
specific elements of it. These included: 


‘Four Rs’, ‘Six Rs’, ‘Seven Rs’, ‘WWW & EBI”, ‘learning partners’, ‘Green Pen’, ‘K-Maths cognitive 
acceleration’ (not a similar approach but involving a lot of reflection and metacognition), ‘self- 
assessment- traffic light or red-amber-green’, ‘Success Criteria’, “writing learning journals”, “learning 
journal time’, “learning objectives combined with Must/Should/Could”, “general things that most 
schools use like AfL techniques” (Assessment for Learning — ‘what helped you today?’, ‘what hindered 


you?’). 


Only two teachers reported that their school did not use any specific approach previously: one of them 
said that ‘a lot of teachers were doing it on their own anyway’ and the other teacher was newly 
qualified when the project started. 


It was apparent from all the school visits that schools had previously established practice around 
encouraging children to be reflective. Most teachers who were interviewed were able to name a 
programme, approach or usual practice in class (see above) where some principles of either 
metacognition, or reflection, or similar had been used. This was apparent for both treatment and 
control classes. While there was no evidence that the ReflectED approach itself was being used in any 
of the control classes we visited, the other forms of reflective practice mentioned above were a part of 
normal practice. There were displays on the walls and teachers in the control classes were making 
use of some of the schemes, e.g. the Green Pen, the traffic light system and Must/Should/Could were 
the ones most in evidence. 


This already existing raft of reflective practices significantly reduces the likelihood that differences 
could be detected between treatment and control arms and the results discussed earlier appear to 
bear this out. However, there was a very strong sense that the inclusion of a systematic process of 
reflection, like that developed by Rosendale, and implemented across the whole school would be 
something that many of the schools would welcome and teachers made it clear that they like the 
systematicity of the ReflectED Approach. 


1 Resilience, Resourcefulness, Reflectivity, Relationships 
? WWW: What Went Well and EBI: Even Better If 
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Fidelity 


Schools reported that overall they adhered to the intended treatment model, some adding that this 
was important as it is part of a research project. Teachers managed to fit in the ReflectED weekly 
lessons, usually having a dedicated slot in their timetable (occasionally moving the ReflectED session 
around in the week, especially if other events were taking place). One school reported that running it 
on a specific day and time was essential because that was when they had access to iPads. Fewer 
schools were more flexible in switching it around within the week: 


“| always make sure if it says half an hour that | give myself half an hour [...] if it’s got to be an 
hour, | switch it to later in the week and do it on a Friday” 


In one case, the teacher delivering the ReflectED lessons was not the same as the regular Y5 class 
teacher. One teacher said that because so many children do other things, it is difficult to find a time 
when all the class is in one place. There were some occasions when too many other things were 
happening at schools (e.g. busy period before Christmas), or because of curriculum constraints, staff 
absence, illnesses, etc. schools reported to being behind schedule with the ReflectED lessons. Nearly 
all schools reported being, or having been in the past, one or two lessons behind and that they either 
dropped a lesson completely or combined it with/squeeze it in one of the other lessons. Sometimes 
something else has to be ‘missed’ in order to ‘catch up’. The Rosendale mentors who worked in the 
schools did not believe that this was a significant problem and agreed that it had taken them a while 
before they were happy with what they were doing in the classes. 


“Honestly... it is quite tricky. With the curriculum and the timetable the way it is, at the 
moment... where are we going to find the time? I've managed to find a slot to teach the 
metacognition lesson every week... On a practical level, it can be quite tricky but when the 
kids do them they do get a lot from it.” 


The Rosendale materials, as well as emails and reminders from Rosendale, were appreciated and 
teachers felt that this contributed a lot in supporting them to follow the recommended structure. Some 
teachers also praised the lessons for being ‘prescriptive’ and easy to follow, with a lot of activities and 
additional ideas. Some schools discussed other issues around the treatment model, such as pairing of 
children, activities, and use of technology. 


The biggest challenge to the fidelity was the time (mentioned by all schools). Finding an extra 30-45 
minutes for the dedicated lessons was very difficult, when there is an already busy programme due to 
the demands of the National Curriculum. Some teachers raised the issue that making reflections also 
takes considerable time, especially if one thinks about a good quality reflection rather than throwing 
together a rushed one. 


Adaptations 


There were not many adaptations reported by schools, as either they did not feel the need to, or, 
because this was a research project and often schools felt inclined to adhere to the recommended 
structure. However, a few schools reported slight adaptations to incorporate previously used 
‘reflective’ practices. In one case, for example, the teacher (an NQT) had felt the need to clarify the 
colours (used in the early stages to be a quick way of summarising their understanding of the lesson) 
and worked together with the whole class to clearly define what each colour meant. Doing this is not 
specified in the ReflectED materials, however the teacher discussed the idea with Rosendale staff and 
it was agreed that this would be beneficial. 


“some of them were saying they were green when they had TA support for the entire lesson, 
so | said ‘yellow, | have succeeded with some support” 
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The same teacher also incorporated the use of marking ladders into the children’s reflections believing 
that it helps them to clarify their success and tie it in with the reflection. 


“if you’ve met all the success criteria, then you’re green; if you’ve met some of them but not 
quite all of them, then at the end of the lesson you will be a yellow” 


One teacher praised the material for being very flexible and easy to adapt if one needs to. However, 
some changes were reported by schools in terms of materials, activities (one school set ambassadors 
to act as mentors to the rest of the class, i.e. those who were particularly good at reflecting), structure 
and so on. In terms of materials overall, a few teachers reported that they may skip little bits according 
to time, or “squashed”/”compressed” some of the sessions in order to catch up: 


“I’ve either not used them, because | haven’t found them useful, or, used them as they were. | 
haven’t made changes” 


Schools overall did not report any major adaptations on materials. Some were already using similar 
activities to those recommended (as discussed above), or added their own individual stories about the 
way that they learned things. In one school a teacher said that she used the ReflectED material to 
make PowerPoint presentations. Another teacher reduced the number of tags that children needed to 
make use of in reflections as they were felt that the length of the list made pupils feel that they have to 
write a lot (which “put them off’). Several schools reported that children who do not enjoy writing found 
it more difficult when it came to writing their reflections down, which occurred later on the ReflectED 
process. 


“...some of the children thought that there was a demand of them to write that much ..... [..-] 
so they don’t have to write a certain amount, they can stick the headings from the book like in 
Evernote, and they can write as much as they like” 


Other schools reported that reflecting supported writing development. One EAL teacher thought that 
the regular requirement to do specific writing during the week had a particularly positive impact on her 
EAL learners (see further detail below in Participant responsiveness). 


In one school, rather than using the pre-prepared sheets provided by the project, children drew a box 
containing the elements for reflection directly into their books and then add colours, emotion words 
and text. When we talked to the teacher about this we were told that this adaptation had been 
discussed and agreed by the Rosendale mentoring team. The children then use these frames before 
and after the lesson, instead of filling in the separate forms, and then when they have time they will 
photograph the reflections and add them into Evernote. The teacher felt that putting the reflections 
directly into the book was more effective and “fluid” than the proformas. A few other schools said that 
children write the reflections (rather than typing them) and they then take pictures to upload on to the 
iPads. Written reflections were also reported being written or glued in the subject workbooks next to 
the relevant piece of work rather than on the Reflection sheets: 


“| just found that if | actually planned [the reflection] into my starter and plenary quite solidly 
and did it in their books — so | asked if that was ok — and | just think it works a bit better 
because they don’t see it as something separate, they see it as part of that lesson, part of the 
learning and | think that cemented it more in their minds” 


In terms of approaches, a teacher said she did “adapt very very slightly”, trying to make them more 
“interactive” and more similar to her regular lessons. 


Sometimes the adaptations reported were not a conscious decision but came as a result of strategies 
invented by the teachers to respond to the challenges of the implementation. For example, one 
teacher reported that sometimes there was a time lapse between the lesson and the reflection and this 
might include the teacher doing marking so that children reflect on what they have learned with the 
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teacher’s comments in front of them to aid the reflection; this, the teacher believes, helps them to 
reflect more effectively especially in areas where they are not doing so well. In another school that had 
large number of pupils who speak English as an additional language, teachers adapted the language 
of all the ReflectED statements (e.g. Blue in this school is: | can work with others). Similarly, a few 
other schools adapted the language slightly, often after discussion with the Rosendale team: 


“some of them were saying they were green when they had TA support for the entire lesson, 
so | said ‘yellow, | have succeeded with some support” 


“I've looked at the lesson plans and | have altered certain things if | think that actually my class 
won't have a clue what I'm on about if | try and do that... so I've altered it... I've simplified it or 
even gone a step further in some cases... I've made it suitable to the children in my class. It's 
easy to make it suitable for the children that | am teaching.” 


Dosage and Quality 


All schools reported that they have been implementing the ReflectED approach from the beginning of 
the academic year, generally managing to run the ReflectED lessons once a week using the teaching 
materials provided. These lessons introduce different aspects of metacognition, but also get the 
children to reflect on what they had done in this specific lesson, as a preparation for further reflection 
on their learning in their regular classes. 


The quantity and quality of reflections in their regular classes was also discussed. In terms of how 
often children were encouraged to reflect per week, some teachers specifically mentioned plans on 


how many reflections children should do per week, e.g. in a case “at least once a week”, “usually twice 
a week”, “at least twice a week”, sometimes “2-3 times weekly”. Another teacher said that the quantity 
of reflections varies also according to a child’s ability, e.g. the higher ability ones may do 4-5 
reflections per week, whilst the lower ability ones would be encouraged to do 2 per week. This is then 
a potential issue for implementation for the children who are the main target of this study and where 
possible children should be encouraged to do similar numbers of reflections. In another school, this 
was more on an ad hoc basis (e.g. if they have a spare ten minutes they will get the ReflectED sheets 
and make use of these to write down reflections on the current or previous lessons). In another school 
the teacher said that if children are not able to finish their reflections at the end of a lesson, then they 
continue the next day. Two teachers stated that they do their reflections in their morning tasks, or in 
guided reading time. The recommendations in the guide provided by Rosendale suggested that 
teachers should find at least two other opportunities in the week for formal reflections as well as 
encouraging the children to reflect independently at other times. Most schools appear to have come 
close to this, although it may need to be made clear in future trials that teachers need to make sure 
that the children do complete this minimum and are reminded that that they should be reflecting at 
other times. 


Strategies to encourage children to do additional reflections were reported, such as a reflections chart 
with names where children mark next to it when they have done a reflection, extra house points, or 
giving prizes to children with the most ticks/reflections. 


In terms of subjects in which reflections happen, there was a variety of responses, with some teachers 
reporting one in literacy and one in numeracy every week, or at least in two subjects per week, etc. 
One teacher mentioned that on an occasion that there was no literacy and numeracy lessons as such 
in the week because of economic awareness week, so she encouraged pupils to reflect in relation to 
that topic. A few teachers mentioned links between the ReflectED materials content and curriculum 
areas (e.g. science); a teacher reported taking the opportunity to link and do reflections around them. 


Quality of reflections, especially against time (see above) was a major issue. Teachers reported 
different strategies towards achieving a good quality reflection, such as missing a little bit of topic time, 
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the teacher recommending a specific aspect of the lesson for the children to reflect on (rather than the 
children having to think about what to reflect on) and using sentence starters. In one school reflections 
were made in subject workbooks across all subjects — originally reflections were made on the sheets 
and then photographed to go on the iPads, but the teacher felt reflections could be more meaningful if 
directly attached to each specific piece of work. Also, some teachers felt that doing an ad hoc 
reflection at the end of a lesson for 5 minutes was enough, whilst others stressed that a good 
reflection might need as much as 20 minutes, especially for some children. 


“We don’t do reflections for every single lesson because quite frankly | would have to stop 
maybe 20 minutes before, we wouldn’t have the time to do it in every lesson... and some 
children they have to think these tricky questions, what went wrong ...” 


Participant responsiveness 


Schools reported that, in general, children had responded positively to the programme and enjoyed 
the ReflectED lessons and learning to be reflective learners. Some teachers explained (or implied) 
that this takes time to achieve: 


“...jit has taken children some time to understand what metacognition is and why they do it, 
time to learn to reflect and especially, to learn to write their reflections. Not all children 
understood what reflection is about and had not been persuaded of its value and why they 
needed it’. 


Several schools reported that some children found writing of reflections challenging. This was 
particularly the case for certain learners, mainly those who did not enjoy writing or had low prior 
attainment. In one school, the teacher said that the children enjoy doing the reflections but they then 
feel disappointed as they realise ‘oh.... | have to write it down now...’. On the plus side, teachers also 
reported evidence of the quality of reflections improving as children developed applicable vocabulary 
to use. Techniques used to help children beginning writing their reflections were sentence starters, 
use of the ‘Learning Objective’ and ‘Steps to Success’ to help structure reflections. 


“| find that those who get the most from it are my children who are the higher attainers already 

. and they know, although there are less things that they find difficult, they know how to 
pinpoint better what exactly it is they find difficult. We had a big barrier at the beginning that 
we're still working on with some of them, some of the lower ability children, where they just 
say ‘I don’t get it, I’m red’ and it’s ‘what don’t you get?’ and they find that really difficult to 
articulate cos they've never had to think about it before ... so it’s good for them to start trying 
to think about it, but it is a battle” 


Other schools indicated that higher ability children have benefited from the understanding that it is 
acceptable to admit that you find aspects of learning challenging and that you can’t always be ‘blue’ 
regarding all tasks. The teachers are suggesting that the children appear to be less reluctant to say 
that they are struggling. 


“They were like, ‘l shouldn’t be red, | shouldn’t ever be red’ because they’ve got this idea of 
themselves. And it’s quite nice as well because they were in colour groups, they automatically 
thought ‘well, I’m a blue, so I’m good at everything’ and the reds tend to think ‘oh, I’m a red, so 
I’m rubbish at everything’. Whereas the colours in ReflectED allow them to be a lot more fluid” 


Similarly, in another school it was felt that the materials and the activities have put some children out 
of their comfort zone. Some of the regular high achievers were challenged by some of the ReflectED 
activities because they did not necessarily do as well with some of the activities as they would 
normally expect to do in class, for example with juggling. For children who do not normally do as well 
in other activities it has helped them show that they can achieve in certain ways and that not everyone 
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is good at everything. For example, a teacher observed that sometimes the lower attaining pupils 
benefitted from the realisation that others struggle on their learning journey and they are not alone: 


“they were very taken by the early term work, in the autumn term you looked at the journey 
you go on as a learner and | think they were relieved and very pleased to see that lots of 
people are struggling that it is an emotional journey and ... | have a couple of children with 
dyslexia and or who just never achieved well at school, never been very successful at school 
and there was this sort of light bulb moment where they went ‘oh, ok it’s not just me, I’m not 
the only one struggling, sitting here wondering what to do’. That was really nice for them at 
that point in the year, just to feel that this is a journey and a lot of them became quite positive 
about their learning as a result, they weren’t giving up. So, it encouraged them to be more 
open. But, as the year’s gone on | feel that they have ... they’re still very young, very immature 
as learners, and they're going to take the ideas that they’ve been given here and I’m sure 
they'll move forward with them and as they go into year 6 they’ll develop them further”. 


In another school, the teacher felt that the approach had benefitted some of the quieter pupils who 
may feel now that “it is ok to not know, and it’s ok to say it; they are brave enough to say it and 
verbalise it’. Other comments concerned the fact that language may be a barrier for some EAL 
learners. One school, as a result, adapted the ReflectED language so concepts were more concrete 
for EAL learners. Another teacher, who said that 95% of the children in her class spoke English as an 
additional language, felt that these children at first did not have the language to explain and it took a 
while for them to understand what it was all about. However, she said the ReflectED lessons 
eventually helped their language develop. 


In terms of SEN, some teachers talked about how students with SEN were adopting ReflectED at a 
slower pace: 


“Really well... the higher ability children took to it straight away and the middle-ability took to it 
soon after... and the lower ability are just starting to pull in with it and understand how to doa 
good reflection on their learning and really get the benefit of it. If that pattern continues, we'll 
hopefully have the SEN up as well. They struggle with it more than the others because, | think, 
they don't have the language to be able to reflect, but the more we use that language, the 
more they are going to get the hang of it. They are definitely starting to realise what things are 
helping them in the classroom.” 


A suggestion was made by a school that further advice on how to work with/differentiate for SEN 
pupils would be useful. 


Several teachers were keen to provide examples of children who were helped by the approach. One 
of them talked about a girl with SEN who with ReflectED could access the words to explain how she is 
feeling. This has helped her overcome her shyness because she has the language to become more 
confident to talk about her emotions. Another teacher talked about a specific boy who had cognition 
delay and reflecting appeared to deepen his understanding and learning. One teacher described the 
extra independence that some of the children develop. This teacher provided an example of a child 
with SEN who is more confident now to use the resources and displays around the room for his 
learning. 


There were also a number of observations concerning the more disadvantaged children. An example 
was given by the teacher whereby they were doing non-chronological reports in English and the 
children asked if they could get their humanities books because they knew that they had material in 
there that they could use. She said that for their school which was in a deprived area, this was both 
very impressive and uncommon. 
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Another area of impact for these children was the added resilience. One teacher said that she has six 
children on Pupil Premium in her class who would previously give up very quickly on any given task 
but the lesson on intrinsic and extrinsic learning was very successful. These pupils realised that not all 
activities have a reward at the end and that there were intrinsic rewards, there were a lot of children 
who were resilient in the beginning but they gained the understanding of the intrinsic reward. 


The role of the iPads? in the study 


Teachers talked about different aspects of technology, including access and the use of iPads, but it 
appeared that only iPads were used in the reflection process. 


There was variability in the use of and attitudes towards iPads and Evernote amongst schools, for 
example, in terms of access to technology as well as in terms of how often and in what ways they 
used it. Use of technology was reported as both a facilitator and a challenge to the teachers’ aim of 
developing reflective learning. For example, whilst a school might have recognised the possibilities 
that the iPads offered in tagging the reflections, at the same time they might have also reported the 
lack of technical support within the school as a challenge to promote it widely as a tool for learning. 


The usefulness of iPads and technology could be a rather controversial issue amongst teachers, 
ranging from teachers very much convinced by it and being enthusiastic about its possibilities to other 
teachers being rather more reserved and even questioning its usefulness. Two such opposing 
opinions are expressed below by two teachers in two different schools: 


“I do find doing it on the tablets a lot easier and almost a lot more purposeful than writing it on 
the sheets and having a big wad of paper everywhere. And the children can write say 40 or 50 
reflections in a year, or whatever the amount, but it’s all just stored in one place and it’s easy 
to find. And with the tags, that’s really useful cos then you can, if you’re looking, for example, 
at a lesson or where a group of children have not performed really well and have tagged 
themselves as red when they felt that they’ve failed in the lesson, then that will just 
automatically group all those pieces of work or all those reflections into one place and | can go 
through it ... if it was a particular lesson or an activity you know, doing compass points or 
whatever it was, | could just type that in as a search and then | could find that lesson and go 
through their work. So, there’s lots of different ways which you can search through Evernote 
and | find that very useful.” 


“... and that was enormously frustrating; it would have been a smoother run if we had a hard 
copy ReflectED notes but my head teacher is very, very keen to go paperless as much as 
possible. She has invested a lot of money on iPads so that we can use iPads which was 
encouraging me to keep going but | have to say, had | been in a different school and we 
hadn't invested so much money on iPads, | would have been tempted just to ditch them and 
go on hard copies” 


The teachers often had different views on the role that technology plays in the whole project, and 
occasionally had specific views or concerns on whether the use of iPads enable or hinder learning 
how to reflect. In one school, for example, the teacher said that children write their reflections on paper 
and a group of children then photograph them and tag them, as “although Evernote is a large part of it, 
it’s not the most important part”. 


3 We use iPads here, because this was the only tablet technology that we saw being used in the schools, but we recognise that Evernote 
can be used on a range of devices including laptops, mobile phone and other tablet types. 
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In another school, the teacher implied that technology might hinder rather than help pupils’ ability to 
make quality reflections. She added that in some cases the use of iPads diverted the attention from 
the reflections: 


“they were more concerned about going on the iPad and taking pictures, now, last week, when 
we had a lesson they wouldn't write a reflection without a picture. | said you don’t have to have 
a picture to write about what you have done or what you want to improve on” 


In another school the teacher said: 


“It's there and it’s good because it engages them and enthuses them but then it can be 
counterproductive as well. But then | definitely think that once they get the knack of knowing 
how to reflect then [...] it will be better” 


Some schools reported that a lot of time was often devoted to the technology issues rather than the 
content of reflections. Such issues ranged from teachers learning to use the Evernote software, 
teaching pupils to use the iPads and the software, slow internet connections, booking in and 
accessing the iPads in schools where overall access to them was restricted, children devoting time to 
taking pictures or tagging rather than thinking about their reflections. However, some of the purely 
‘technological’ barriers were more of an issue to start with and these schools had arrived at a better 
level of use of the technology by the time we visited them to observe classes and to interview 
teachers. Technology as a challenge was often reported by schools as one that can be overcome with 
time, support, training and technical advances. Teachers, for example, often reported that the 
technology had been more of a challenge towards the beginning of the project, but both themselves as 
teachers and the pupils, soon became familiar with the Evernote through practice and continuous 
support. One teacher said that the children were getting “distressed” and “frustrated” initially, although 
this was not now so much of a problem. As tablet technologies become more embedded into the 
school system in the UK, good internet access is a standard feature of a school’s digital infrastructure 
and teachers become better aware of the role that tools like Evernote can play, then these issues will 
disappear, but in the short term they will remain a barrier to implementation. 


In terms of the approach developed by Rosendale the use of the iPads was a key feature of the 
implementation, however quite a few schools suggested that if they went forward with a whole school 
implementation of the project that they may well not make use of the Evernote software to implement 
the project. It seemed that many of the teachers had not grasped the possibilities that the storing of 
the reflections would open up for them in terms of the monitoring of progress across the school. 
Teachers that we talked to at Rosendale made it very clear that they thought that the use of iPads was 
very beneficial for understanding children’s progress in classes and used the data provided by the 
children actively in giving feedback. This was not something that appeared to have been so well 
appreciated in the treatment schools. 


In terms of numbers of iPads, schools reported as few as 5 iPads for the whole class (at the beginning 
of the academic year) to as many as 16 iPads for 30 students. Teachers without many iPads available 
used strategies such as having pupils write reflections on paper/whiteboards before they take pictures 
and tag them, carousel use and using iPads during guided reading time to encourage as many 
children as possible to have a go. They all agreed that an iPad per child would help. The most usual 
practice was one iPad per table or one iPad per pair of children. One teacher whose class had one 
iPad per table said that they also had a spare one to allow any child who wanted to reflect 
spontaneously to do so. One teacher acknowledges: “We haven't used the technology to its fullest 
extent as of yet. | think the limiting factor is that we don’t have an iPad for each child. If every child had 
an iPad we would maximise the benefit of recording everything on Evernote.” One teacher suggested 
the use of technology at an earlier stage in pupils’ school life so that pupils can more develop 
technological competence before they start using Evernote. Another problem reported by at least two 
schools was that iPads were not available throughout the week and had to be timetabled and could 
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therefore only be used at specific times (for example, in one school, the only time that that the tablets 
are used with Evernote is in the ReflectED lesson). 


Limitations of the software were also very occasionally mentioned by the interviewed teachers. In one 
school, the teacher referred to the software as: 


“unforgiving! If you mis-tag by one letter, the search then doesn’t bring it out. So you have to 
be very precise in your tagging and labelling of lessons. We got that sorted and we’re moving 
on nicely. We don’t have enough reflections to make that an issue. When you’ve got hundreds 
of reflections, then you can go back and look at how the reflections make a difference.” 


In summary, the main challenges raised by teachers in relation to the use of technology and iPads 
were in terms of: 


e technological challenges such as internet connection breaking down, passwords not working; 
e number of iPads per group of learners 

e limitations of the software; 

e possibly diverting attention away from the quality of reflections. 


However, all schools acknowledged that the use of technology was received very well by the majority 
of pupils, and the opportunity to use technology often motivated them to note their reflections. Focus 
group discussion with pupils confirmed these reports. Overall, children spoke with enthusiasm and 
often with a sense of pride on what they have learned to do on the iPads, and they were often 
impressed with the possibilities that the Evernote offered (e.g. searching to find reflections by the 
month when these were included, or by topic; being able to see what their peers thought about a 
classroom activity they had ReflectED on). 


Children appeared to be very keen to use the iPads and Evernote, found it easy to use, and liked 
aspects such as taking pictures of their reflections or other aspects of their work. However, reporting 
on the ease of using iPads and particularly the Evernote software varied between pupils in the same 
school and between schools (some teachers reported how quick and easy it was for pupils to learn to 
use the tools and others how challenging it was and how long it took children to get used to them). 
Teachers also reported asking children to firstly write their reflections and then take a picture and 
upload them, rather than typing straight into the tablet (mainly to overcome the limitations of children’s 
typing, availability of iPads, or time constraints). For example, two schools reported that they selected 
a small group of pupils to photograph and upload the reflections for the whole class as she felt that 
many of the children were not competent enough to manage the technology. This strategy also 
resolved the challenge of limited accessibility to iPads. Pupils occasionally raised concerns over 
security. For example, some pupils worried about protecting their privacy, and felt scared that others 
might be looking at their reflections or deleting them. 


Formative findings 


The main confounding factor that became evident as the trial continued was the level of practice 
around reflection that was already in evidence in the schools. Most of the schools we visited during the 
process evaluation had some form of reflective practice already on-going prior to the commencement 
of the trial and they often appeared to be adding the RelectED approach to this practice. This is likely 
to have had an impact in preventing any significant effect size and future trials need to consider how 
this factor might be mitigated. The use or not of the Evernote software for recording the reflections 
was also an issue that got raised quite a lot by the teachers in the interviews and focus groups and 
how this might be handled in future studies is discussed below. 


However, although they were positive towards the ReflectED approach, it was often applied on top of 
previous approaches, or adapted slightly to fit in with them, or sometimes combined with previous 
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practices. For example, in one school, whilst ‘What Went Well & Even Better If (WWW & EBI’)’ was 
still used, children got extra house points for using ReflectED reflections instead of ‘WWW & EBI’. Two 
schools mentioned that they chose to adopt ReflectED in full owing to the fact that this was part of a 
trial project. 


All teachers we spoke to had something positive to report in relation to ReflectED compared to 
previous approaches (e.g., the dedicated ReflectED sessions, or, the quality and intensity of 
reflections). Most teachers felt that they would like for their school to continue using the approach, 
although this was often dependant on a number of factors (e.g. evidence of impact, on condition to be 
rolled out in whole school, to start at foundation level, to have more allocated time on it, to have more 
iPads or other resources, to adapt it to suit their needs etc.). Some teachers (four of the 19 
interviewed) were more sceptical on the future use of ReflectED or not totally convinced by it. 
However, only one of the teachers who were interviewed said that she would not like to use the 
lessons as they are in the future, even if the school decided to continue the approach. At least three 
quarters of the teachers said or implied that they would take on board aspects of the approach and 
possibly use elements of it even if their school decides not to roll our ReflectED across the whole 
school. Several teachers reported that delivering ReflectED across the whole school would be more 
sustainable and easier to implement than delivering it to just one class, as this would enable the 
school to take decisions overall in terms of some of the challenges. In addition, running it as a whole 
school would enable children to learn to reflect at a younger age, and have time to develop their 
metacognition throughout primary school. Several teachers also commented that it would be easier 
running it second time around. 


“Yes... definitely... | think it's going to have a fantastic impact on them - particularly here, in the 
area we are in. | think if they started this in Key-Stage 1 and brought it all the way up through 
the school it would be amazing to see what they could achieve.” 


It is also interesting to note that Rosendale Primary School found ReflectED to be most effective when 
it is delivered across the whole school. An important feature of ReflectED for the Rosendale teachers 
has been the development and evolution of the process in collaboration with the children. They see 
ReflectED as a process that is evolving and not static. A second issue is that ReflectED at Rosendale 
begins in the infant phase, which clearly impacts on their success later on. In the case of this trial it 
must be recognised that ‘going in cold’ in year 5 will have implications for the dosage compared with a 
model which would start lower down the school. 


Control group activity 


We found no evidence that the control groups in the schools were affected by what was happening in 
the treatment classes and there was little evidence of any negative feeling towards the trial. One 
teacher had been keen to be involved in the treatment, but had been picked as a control teacher. 


We saw no evidence of contamination of the control classes. There was none of the material from the 
ReflectED approach in any of the control classes we visited, nor did we see any of the proformas 
being used in the children’s workbooks. There was evidence of metacognitive work going on as we 
have described above, but none of the specific methodology of the ReflectED approach. 
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5. Conclusion 


Key conclusions 


1. Pupils who participated in ReflectED made an average of four months’ additional progress in 
maths compared to pupils who did not. 


2. Pupils who participated in ReflectED made an average of two months’ less progress in reading 
compared to pupils who did not. 


3. The findings for the schools in this trial have moderate to high security. However, the analysis 
conducted suggests that we cannot conclude from this trial alone that the intervention would 
have a similar impact in other schools. 


4. Most schools were already teaching metacognitive and reflective skills similar to those 
encouraged by ReflectED. This might have limited the additional impact that ReflectED had on 
teachers’ practice and pupils’ outcomes. 


5. Teachers suggested that ReflectED would work best as a whole-school programme, and that 
they could deliver the programme more effectively after the first year of delivery. Future research 
could examine the impact of implementing ReflectED across all year groups in the school and 
allowing more time for the programme to become embedded. 


Interpretation and generalisability 
The aims of the study were to: 


1. assess the impact of the ReflectED approach on children’s academic attainment in 
mathematics, 

2. determine if the ReflectED approach also produced differential gains for children eligible for 
free school meals (FSM), 

3. assess the impact of the ReflectED approach on children’s academic attainment in reading 
and attitudes towards mathematics and reading, 

4. explore attitudes towards the introduction of the ReflectED approach, and 

5. identify process-related issues that might help us to better understand our main impact 
findings. 


Aims one to three were addressed using a school-based randomised trial design with class-level 
randomisation. Pupils who received ReflectED made more progress in maths than pupils who did not, 
but the impact estimate was not statistically significant. Pupils who were eligible for FSM made more 
progress if they received the intervention but this effect size was not statistically significant. Pupils who 
received ReflectED made less progress in reading than those who did not, but this was not statistically 
significant. Pupils in the intervention group develop a more positive attitude towards maths, but this 
was not statistically significant. Pupils who received the intervention developed a slightly worse 
attitude to reading, but this was not statistically significant. 


In terms of aims four and five, the ReflectED approach was viewed positively by most of the teachers 
who were interviewed, or who attended the focus groups. Evidence from the process evaluation 
suggests that teachers had mixed views as to which groups this approach was most effective for, with 
EAL, SEN, lower and upper ability cited by different teachers. One of process related findings that may 
have limited the impact of the programme was the emphasis already being put on reflective practices 
in the schools. The process evidence shows that schools had already begun a process of 
implementation of reflective systems so ReflectED itself may not have had an additional impact on 
practice. There was also a perceived lack of time for implementation, particularly around the 


development of the skills to record of the reflections using the Evernote system and for the teachers to 
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be able to make use of the databases of reflections to be able to further support the children’s 
development. It was also suggested that a whole school approach would be the most effective way to 
implement an effective reflective approach. 


Schools did not seem to perceive the recording of the reflections on to Evernote as important and 
focused much more on supporting the children to reflect generally. The Rosendale team see the 
Evernote process as a central part of the ReflectED approach, giving both children and teachers an 
insight into learning processes. Being able to search through the database for indications of progress, 
or weakness in a particular area was felt to be a powerful tool by the Rosendale team, but not 
appreciated in the same way by the teachers in the trial schools. Future research could examine how 
to support teachers and children to use Evernote. 


Limitations 


A key issue that needs to be addressed would be how to take account of existing metacognitive 
schemes already in play in many schools. There was clearly significant interest in metacognition in the 
schools shown in the study. 


A second issue is the level of dosage for the intervention. Many of the teachers talked about the time 
that they were able to spend on the intervention and it was evident from the discussions with the 
teachers that they felt that they had only really begun to be able to effectively apply ReflectED in the 
classes after one year. This suggests that future trials may want to allow longer time for the 
intervention to be embedded before anticipating it to have an effect on pupil attainment. 


Third, in the final analysis, the study was arguably underpowered. The MDES of 0.4 at the analysis 
stage means that our modelling was insensitive to the effect size observed for our primary outcome 
(Hedge’s g = 0.3; hence, the marginal non-significant trend at p=0.08). On the other hand, an effect 
size of 0.3 may be seen as meaningful, being equivalent to 4 months’ additional progress in 
mathematics. However, the power limitations noted above mean that we cannot generalise from this 
finding to the impact on other schools, and thus a future (larger) study powered to be sensitive to 
smaller effect sizes (e.g. MDES = 0.2) would be necessary to resolve this uncertainty. 


Future research and publications 


Further research into metacognition would appear to be warranted considering the confounding issues 
with the control condition (i.e. teachers using metacognitive strategies) the underpowered nature of 
the final sample, and the strength of the positive feelings of teachers towards metacognition. 


There was interesting evidence from the process data that children in minority groups were beginning 
to perform better in classes. EAL children were particularly mentioned in this regard, but also children 
with SEN. There was also clearly a strong desire for metacognition to be part of regular school 
practice and for it to be used across the whole school. Teachers in the process study believed that 
metacognition has value and so do previous studies. The interest in materials produced by Rosendale 
was very high and there were a number of suggestions that ReflectED was more suited to a whole 
school approach. Working to find schools that were not currently using any kind of reflective practice 
might be one way forward in terms of finding more significant results, as would finding ways of 
increasing the power for all pupils primarily, but also with respect to smaller minority groups in some of 
the schools. 


One area that has not yet been explored is the considerable database of reflections produced by the 
children both at Rosendale and other schools. These could prove a really useful source of data to 
show how the learners progress and in what ways. 


Education Endowment Foundation 35 


ReflectED 


We intend to publish the findings from this study in both practitioner and academic periodicals. We will 
go back to Rosendale school and work with them on producing joint outputs to be used in conference 
presentations and also for peer reviewed journals. 


Further research could include: 


1. 


2. 


Look into how integral Evernote is to ReflectED and to either improve its use, or to find 
alternative ways for schools to record 

Increase power by sample selection 

Consider how to accurately record dosage of reflection/metacognition in ReflectED and in 
control condition 

Produce a systematic literature review of the use of different reflective practices in education 
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Appendix 1: Parental Consent forms 


ReflectED PROJECT 


Weare writing to you to tell you about an exciting project that the school is involved in which is aimed at 
supporting children to better understand how they learn. The programme is called the ReflectED 
approach. 


The ReflectED programme is for all children and helps them to understand better how they learn, how 
they can identify when they are learning well and when they need to make changes, or to seek help. This 
process is sometimes referred to as ‘Learning to learn’, or more technically as ‘Metacognition’. Our 
research project is looking at the implementation of this process and is supported by a set of materials 
designed by a team at Rosendale Primary School, West Dulwich, London, who have been working with the 
ReflectED approach since 2012. The research we are conducting will help us to understand if the 
ReflectED approach works for children in Year 5 (2014-2015) and then will be introduced into other 
years, if the schools are happy with the findings. The project is funded by the Educational Endowment 
Foundation (http://educationendowmentfoundation.org.uk). 


We are writing to you to explain the role of your child in the project. Our plan is to collect data from 
children’s Key Stage 1 SAT tests in the next few weeks and then all of the children will take a short 
computer-based test at the end of the school year, around June. Some of the children in about 50% of the 
schools will also be selected to be part of focus groups following a visit to the school to observe and 
interview teachers. The teacher or another member of staff familiar to the children will be present during 
these focus groups. 


Who will conduct the research? 


The research will be conducted by Dr. Gary Motteram and other staff in the Institute of Education, 
University of Manchester, Oxford Road, Manchester M13 9PL. 


Title of the research: 
"ReflectED" 
What is the aim of the research? 


Our main aim is to examine the impact of the ReflectED Approach on children’s progress in maths and 
reading during one academic year. 


Where will the research be conducted? 
Primary schools in a number of LEAs in various parts of the country. 
What is the duration of the research? 


The overall project runs from Sep 2013 until August 2015. The schools that implement the ReflectED 
Approach will do so from September 2014 to July 2015. 


Why have I been approached? 


We are writing to you because your child’s school is taking part in the ReflectED Project and the School 
will be implementing this approach from September 2014 to July 2015. 


What would my child be asked to do if I agreed for her/ him to take part? 
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All the children in participating schools will be involved in one short computer-based test, which will take 
30-40 minutes, and will take place in June 2015. In some cases children will be invited to take part ina 
short focus group to talk about their experiences of the ReflectED Approach. The results of the test will be 
compared to the data collected from the KS1 SATs tests to look for differences, essentially to try to 
discover ifthe ReflectED Approach has had an impact. The focus groups will help us to understand why 
change has or hasn’t occurred. 


What happens to the data collected? 


The data will be analysed by the research team at the University of Manchester. We will write a report 
based on our analyses for the Educational Endowment Foundation. It is also likely that we will write 
articles for academic journals based on the project findings. Finally, it is possible that we will write a book 
chapter about the research. In all publications and reports data will be presented anonymously. 


How is confidentiality maintained? 


All data provided will be treated as confidential and will be completely anonymous. Identifying 
information (e.g. pupil names) will only be used in order to match responses about the same individual 
from different respondents (e.g. teachers and pupils) and across different times (e.g. at different points in 
the year). After this matching process is complete, all identifying information will be removed. 


All the data will be stored on secure, password protected computers to which only senior members of the 
research team have access. 


Disclosure and Barring 


Every member of our research team has undergone a Disclosure and Barring check at the Enhanced 
Disclosure level. 


Contact for further information 


If you would like any more information or have any questions about the research project, please 
telephone or email: 


Dr. Zeynep Onat-Stelma, Manchester Institute of Education, University of Manchester, Oxford Road, 
Manchester, M13 9PL - Tel: 0161 275 3901 - Email: zeynep.onat-stelma@manchester.ac.uk 


Also, please see the EEF website for further details about the project: 
http://educationendowmentfoundation.org.uk/projects/ReflectED-meta-cognition-rosendale-primary- 
school/, or check at the Manchester Institute of Education website: 
http://www.seed.manchester.ac.uk/subjects/education/ 


If you ever wish to make a formal complaint about the conduct of the research you should contact the 
Head of the Research Office, Christie Building, University of Manchester, Oxford Road, Manchester M13 
OPL. 
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CONSENT FORM 


An information sheet is attached to this form. Please read it carefully before making a decision about 
letting your child take part in the data collection and the focus groups. 


If you are willing for them to take part then you do not need to do anything at the moment. You will be 
sent further details about the focus groups in the near future. 


If you decide that you do want your child to take part, then you need to complete the opt-out consent form 
below and return it to Dr. Zeynep Onat-Stelma, Manchester Institute of Education, University of 
Manchester, Oxford Road, Manchester, M13 9PL 


Tel: 0161 275 3901 


Alternatively, Dr. Onat-Stelma can be contacted by telephone on 0161 275 3901 or email at zeynep.onat- 
stelma@manchester.ac.uk. If you do not wish to participate please let us know by XXXXXXxX. 


Finally, please also remember that even if you do decide to let your child take part now, you are free to 
change your mind at any point in the study. 


Please tick in the box next to the first statement if you want your child to not take part in the project 
overall, or in the second box if you don’t want your child to take part in just the focus groups. 


I do not wish my child to participate in any part of the project. 


I do not wish my child to participate in the focus group strand of the ReflectED project. 


My details are as follows: 


My name 


My child’s name 


Name of my child’s school 


Signed: Date: 


Please return this form to Dr. Zeynep Onat-Stelma, Manchester Institute of Education, University of 
Manchester, Oxford Road, Manchester, M13 9PL. 
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Teacher Consent form 


INFORMATION SHEET FOR TEACHERS BEING INTERVIEWED: ReflectED PROJECT 
As you know your school is involved in an exciting project about supporting children to better 
understand how they learn, called the ReflectED approach. The research we are conducting in 
conjunction with the team Rosendale Primary School in West Dulwich will help us to understand 
if this approach works for children in Year 5 and, if it is, it will then be introduced into other 
years, if the schools are happy with the findings and are convinced by the benefits. The project is 
funded by the Educational Endowment Foundation. 


We are writing to you to explain your role as a teacher in the research process. We will collect 
your views and those of some of the children in your class. 


If you would like any more information or have any questions about the research project, please 
telephone Dr. Zeynep Onat-Stelma on 0161 275 3901 or email her at zeynep.onat- 
stelma@manchester.ac.uk 


Who will conduct the research? 


The research will be conducted by Dr. Gary Motteram and other staff in the Institute of 
Education, University of Manchester, Oxford Road, Manchester M13 9PL. 


Title of the research 
The ReflectED Approach 
What is the aim of the research? 


Our main aim is to examine the impact of ReflectED Approach on children’s progress in maths 
and reading during one academic year. 


Where will the research be conducted? 

In primary schools in a number of LEAs around the country. 

What is the duration of the research? 

The project itself runs from September 2014 until August 2015. The schools that implement the 
ReflectED Approach will do so from September 2014 to July 2015 and this is the period during 
which data is collected. 

Why have I been chosen? 

We are contacting you because your school is taking part in the ReflectED Project. Classes in the 
participating schools have been randomly chosen to (a) implement the ReflectED Approach for 
one year, (ReflectED classes) or (b) continue as normal (comparison classes). We will be 


collecting data in all of the schools. 


What would I be asked to if I took part? 
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All teachers introducing the ReflectED Approach in their classes will be invited to be part of 
focus groups towards the end of the trial and at least 50% of all of the participating teachers will 
be observed in one lesson and invited to participate in a short interview about the on-going 
implementation. The observation will only relate to the implementation of the ReflectED 
Approach and this is why we are approaching you. 


The interview will last a maximum of 40 minutes. 
What happens to the data collected? 


The data will be analysed by our research team at the University of Manchester. We will write a 
report based on our analysis for the Educational Endowment Foundation. It is also likely that we 
will write articles for academic journals based on the project findings. Finally, it is possible that 
we will write a book chapter about the research. In all publications and reports data will be 
presented anonymously. 


How is confidentiality maintained? 

All data provided will be treated as confidential and will be completely anonymised. Identifying 
information will only be used in order to match responses about the same individual from 
different respondents (e.g. teachers and pupils) and across different times (e.g. at different 
points in the year). After this matching process is complete, all identifying information will be 
destroyed. The website that houses the survey will be completely secure and password 
protected. All survey data will be stored on a secure, password protected computer to which 
only senior members of the research team have access. 


Disclosure and Barring 


Every member of our research team has undergone a Disclosure and Barring check at the 
Enhanced Disclosure level. 


Will I be paid for participating in the research? 
We are not able to offer any payment or incentive for participating in this study. 
Contact for further information 


If you would like any more information or have any questions about the research project, please 
telephone or email: Dr. Zeynep Onat-Stelma, Manchester Institute of Education, University of 
Manchester, Oxford Road, Manchester, M13 9PL 


Tel: 0161 275 3901; Email: zeynep.onat-stelma@manchester.ac.uk 

Also, please see the EEF website for further details about the project: 
http://educationendowmentfoundation.org.uk/projects/ReflectED-meta-cognition-rosendale- 
primary-school/, or check at the Manchester Institute of Education website: 


http://www.seed.manchester.ac.uk/subjects/education/ 


What if something goes wrong? 
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If completing being involved in the research makes you worry about any of your pupils’ 
wellbeing then you should speak to your school’s safeguarding and child protection officer in the 
first instance. 


If you ever wish to make a formal complaint about the conduct of the research you should 
contact the Head of the Research Office, Christie Building, University of Manchester, Oxford 
Road, Manchester M13 9PL. 


Education Endowment Foundation 44 


ReflectED 


CONSENT FORM 


If you are happy to participate in the interview please complete and sign this consent form 


Please 
Initial Box 


1. | confirm that | have read the attached information sheet on the above study and have had the 
opportunity to consider the information and ask questions and had these answered satisfactorily. 


2. understand that my participation in the study is voluntary and that | am free to withdraw at any time 
without giving a reason. 


3.  lunderstand that the interview will be audio-recorded. 


4. | agree to the use of anonymous quotes. 


ol 


| agree that any data collected may be passed to other researchers in the team. 


a 


| agree that any data collected may be published in anonymous form in academic books or journals. 


7. | agree to take part in the above project. 


Name of participant Date Signature 
Name of person taking consent Date Signature 
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Appendix 2: MLM models (Primary Outcome) 


Table 11: Standard deviation used to standardise Z scores (Hedges G) 


Standard 
Deviations 
Mathematics InCas 21.24 
Reading InCas 15.46 
Attitude to mathematics 37.99 
InCas 
Attitude to reading InCas 38.17 


Table 12: Raw and prior attainment adjusted ICC 


Prior attainment adjusted 


Raw ICC 


[ores 

Mathematics InCas 0.28 0.34 

Reading InCas 0.28 0.31 

Attitude to mathematics 0.04 0.06 
InCas 

Attitude to reading InCas 0.04 0.04 


Table 13: InCAS mathematics scores clustered at class level with and without imputation 
(complete cases) 


InCAS 
Mathematics InCAS Mathematics 
Toxo) =) Score (multiple 


(complete imputation) 
cases) 
Beta | S.E. Beta SE. 


cons -3.09** | 0.45 |) -2.52** 0.56 
pidiuchieaes das 0.30 017 0.27 0.19 
control) 
KS1 maths score 0.19** | 0.03 0.15** 0.03 
FSM (Ref: no) -0.13 | 0.19 -0.21 0.21 
Level: Class 
cons/cons 0.48** | 0.13 0.59** 0.16 
ReflectED/cons -0.03 | 0.08 -0.06 0.10 
ReflectED/ReflectED | 0.00 | 0.00 0.00 0.00 


Please note that ** stands for 5% significance level. 
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Table 14: Subgroup analysis (FSM only) - InCAS mathematics scores clustered at class level 
with and without imputation (complete cases) 


oe 258 0.6 2.54** 0.65 
ReflectED (Ref: 

control) 0.14 0.2 0.07 0.21 
KS1 maths score 0.15** 0.04 | 0.14** | 0.04 
Level: Class 

cons/cons 0.8** 0.21 | 0.84** | 0.22 
ReflectED/cons -0.15 0.12  -0.17 0.13 
ReflectED/ReflectED | 0 0 0 0 


Please note that ** stands for 5% significance level. 
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Table 15: Secondary outcomes clustered at class level with and without imputation (complete cases) 


InCAS reading 
scores 
(complete 
cases) 


Beta 


BS} = 


InCAS 
InCAS reading PNatie(e (om ce) 
scores (with mathematics 


Tan) eeiectareya)) (complete 
cases) 


Beta CHS. Beta SE 


InCAS Attitude 
to mathematics 


(with 


any eleneui(eyay) 


Beta 


SE 


InCAS Attitude to 


ig=r-Vel tare] 


(complete cases) 


Beta 


SE 


ReflectED 


InCAS Attitude 
com csr-Conlalem QuuUlag] 
any eLuic:viceya)) 


Beta S.E. 


cons 158 | 0.50 | -1.52* 0607-071 064 | -0.71 | 074 | -086 | 045 | -0.74. 0.60 
per oa 0.15 | 0.23 | -0.05 | 0.228) 0.23 | 0.24 | 035 | 0.26 | -0.09| 0.19 | 007] 022 
FSM (Ref: no) 0.42 | 0.23) -043  0.231| 0.12 | 0.27 0 0.32 064 0.21 042 0.24 
KS1 Reading Score 0.13** 0.03. 0.12** (0,033 0.06* 0.03 0.05 0.03 
KS1 Maths Score 0.04 0.04 0.03 0.04 
Level: Class 
cons/cons 1.03" 0.27. 0.88" 0.222 0.99% 026 099% 0.28 062% 016 062 0.18 
ReflectED/cons 0.24 | 0.15 | -0.21 |0.126| -0.08 | 016 | -005 | 018 | -0.02 | 0.11 0.03 0.13 
ReflectED/ReflectED 0 0 0 0 0 0 0 0 0 0 0 0 


Please note that ** stands for 5% significance level. 
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Appendix 3: Padlock rating 


Design Power Attrition* 


5a 
MDES < 0.2 0-10% 


----4 Adjustment 


4a Fair and clear quasi- for Balance 


experimental design for 
comparison (e.g. RDD) with 
appropriate analysis, or [-] 
experimental design with minor 
concerns about validity 


3a Well-matched comparison 
(using propensity score 
matching, or similar) or 
experimental design with 
moderate concerns about 
validity 


MDES < 0.4 21-30% 


Weakly matched comparison or 
experimental design with major | MDES <0.5 31-40% 


Adjustment 
flaws 


for threats 


----7  tointernal  |----- 


Comparison group with poor or validity 


no matching (E.g. volunteer MDES < 0.6 51-50% 
versus others) [-] 


No comparator MDES > 0.6 <50% 


e Initial padlock score: lowest of the three ratings for design, power and attrition = 4 padlocks 
e Reason for adjustment for balance (if made): n/a 

e Reason for adjustment for threats to validity (if made): n/a 

e Final padlock score: initial score adjusted for balance and internal validity = 4 padlocks 


*Attrition should be measured at the pupil level, even for cluster trials. 
Notes 


e Well-designed trial with randomisation at class level 

e 15.5% attrition 

e The two groups are balanced at baseline 

e There was some suggestion from peer reviewers that there should be an adjustment for threats to 
validity as the tests were not delivered blinded, but we think this threat is minimal as schools would have 
had to actively manipulate the results of one class in the year group, which seems unlikely 
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Appendix 4: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. More information about the EEF’s approach to cost evaluation can be found here. 
Cost ratings are awarded as follows: 


Cost rating Description 


ELEEEE Very low: less than £80 per pupil per year 
ELEELEEE Low: up to about £200 per pupil per year. 


EEEEE High: up to £1,200 per pupil per year 
ELELEEE Very high: over £1,200 per pupil per year 


ELEEELEE Moderate: up to about £700 per pupil per year. 
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